Efficient Computation of Unconditional Error Rate Estimators for Learning Algorithms and an Application to a Biomedical Data Set Master Thesis

We derive an unbiased variance estimator for re-sampling procedures using the fact that those procedures are incomplete U-statistics. Our approach is based on careful examination of the combinatorics governing the covariances between re-sampling iterations. We establish such an unbiased variance estimator for the special case of K-Fold cross-validation. This estimator exists as soon as new observations are added to the original sample, and we specify how many additional observations are necessary. Thus we make re-sampling procedures comparable. We make no assumptions on the underlying distribution and we take the covariances between re-sampling iterations into account. Beyond that we show an approach to find a re-sampling design with minimal variance for a fixed size of learning sets. We empirically show the existence of designs with smaller variance than repeated cross-validation. We systemically compare with the complete U-statistic, the leave-p-out estimator. Our examination is completed by an application to micro-array data.